Modern computer systems function under the control of an operating system—a set of computer-readable instructions that control low level functions within the computer system (for example, memory allocation, input and output distribution, interrupt processing and job scheduling). A well-known example of an operating system for personal computers is MICROSOFT'S WINDOWS™ family of operating systems. Although critical to the operation of the computer system, much (if not most) of the operating system's functioning is hidden from users of the computer system, who instead tend to interact with application programs to perform tasks such as composing documents or sending and receiving e-mail, etc.
Application programs are generally what a user thinks of when he or she thinks of a computer program. That is, a set of computer-readable instructions that allow the computer system to perform a specific task, such as word processing or the like. Application programs make use of operating systems to manipulate the resources of the computer system in desired ways to carry out these tasks. In doing so, the application program makes requests of the operating system functions through an application programming interface (API). When an application makes such a request, it is said to issue a “call” for the desired function.
The API defines how the application program must present its request (or call) for the desired service and the information (or “arguments”) that must be provided to the operating system in order for the request to be carried out. Modern application programs include literally thousands of calls for things such as displaying characters on the screen of a computer system, and reading and writing information from/to the computer system's memory. When the requested function has been completed, the operating system returns control to the application program (and, in some cases, may also return a value for further processing by the application program).
Occasionally, it is desirable to change the manner in which a particular application program interacts with the operating system (or another target computer program of the application programs' requests), or to extend or monitor the behavior of the application program in some fashion. So-called “API hooking” is a programming technique that can be employed to carry out these tasks. So-called “hooks” are examples of computer program execution stream modifications that can be used to redirect a set of computer-readable instructions from its customary path (e.g., to a replacement function). This replacement function typically will perform some manipulation of the arguments passed as part of the call before transferring control back to the called API. Likewise, the replacement function can be used to manipulate results returned from the called API before passing them back to the application program. Thus, API hooking allows for monitoring and/or interception of calls for services made by an application program during its execution.
API hooks can be introduced in a variety of ways, either at the source of the call (i.e., the application program) or the target thereof (e.g., the operating system process being called). A particularly powerful method of API hooking involves injecting a dynamic link library (DLL) into the address space of the target program and patching the API with a branch (i.e., JMP) instruction. A DLL is a file containing executable instructions (in the form of subroutines and functions) that can be loaded and executed by a computer program while that program is running. The JMP instruction is simply a mechanism by which to transfer program control to a desired point in an instruction stream.
As shown in FIG. 1, when the above-described procedure is used, calls from an application program 10 to the intercepted API 12 are redirected by a hook stub 14 to a hook function 16 inside the injected DLL. The hook function 16 can then perform the desired manipulation (e.g., such as inspection or modification of arguments) before, after, and in some cases instead of passing control to the original API 12. The hook stub 14 is used is to prepare the stack frame (a portion of a call stack for the computer system—the call stack is used to keep track of the point to which each request for services should return when the request has been completed) and to keep a reference count of the number of concurrent threads (individual program flows) running inside the hook function 16.
This sort of API interception can be used with both on-demand software (software applications that are used on an as-needed basis but which are not installed on the subject computer system) and persistent software (software applications installed on the subject computer system), to control the behavior of application programs and provide additional context information concerning same. Nevertheless, a technical challenge is introduced when an on-demand application is terminated, or a persistent software application is uninstalled, and that application needs to clean up its DLLs from the subject computer system. The Windows operating system prevents an injected DLL (containing hook functions) from being deleted until it is ejected from the address space of all applications. But, hook functions such as those described above typically chain to the original APIs that can block execution for prolonged periods of time. Hence, as shown in FIG. 2, ejecting the DLL (and the associated hook function) from the address space of the target program would result in a crash (i.e., an exception) when the call returns back into the unloaded DLL code (a situation represented by the dashed lines).
In addition to the above, ejection of the “hooking DLL” may take place at an appointed time or when an application program that performed the initial injection of the subject DLL (a so-called controlling application) terminates. Or, there may be other scenarios that arise that lead to explicit ejection of the subject DLL or a self-ejection thereof. Hence, what is needed is a technique for safely ejecting a DLL containing one or more API hooks in such situations.